Some continuous analogs of expansion 
in Jacobi polynomials and 
vector valued hyper geometric orthogonal bases 

Neretin Yu.A. 

Abstract. We write spectral decomposition of the hypergeometric differential 
operator on the contour Re 2: = 1/2 (multiplicity of spectrum is 2). As a result, we 
obtain an integral transform that differs from the Jacobi (or Olevsky) transform. We 
also try to do a step towards vector-valued special functions and construct a 3-F2- 
orthogonal basis in the space of functions having values in 2-dimensional space. This 
basis is lying in an analytic continuation of continuous dual Hahn polynomials with 
respect to number n of a polynomial. 

In Addendum, we discuss C'^-value analogs of the Meixner-PoUachek orthogonal 
system and also perturbations of Laguerre, Meixner, and Jacobi polynomials. 

§1. Representation-theoretical motivation and formula- 
tion of results. 

1.1. Continuous analogs of expansion in Jacobi polynomials. The 

present work is an counterpart on the level of special functions of Molchanov's 
paper |17| on tensor products of unitary representations of the group SL2 (M) . 

In the classical analysis, there are well-known the expansion in Jacobi poly- 
nomials and also its continuous analogue. The latter is the index hypergeo- 
metric transform of H.Weyl, [33] (it is also called by Olevsky transform |23|. 
Jacobi transform, generalized Fourier transform, see Koornwinder's survey 
see also J^li ESI)- These classical constructions have a transparent and 
important representation theoretic interpretation (see j32j). Spherical functions 
of the projective spaces 

0(n+l)/0(n)xO(l), U(n-f 1)/U(n) x U(l), Sp(n + 1)/Sp(n) x Sp(l) (1.1) 

over M, C and the quaternion field H are the Jacobi polynomials P^''^ for some 
special values of the parameters a, (3. The theorem about decomposition of 
on these spaces into a direct sum of irreducible representations is a corollary of 
the theorem on expansion of a function into a series in Jacobi polynomials. 
For hyperbolic spaces 

0{n, l)/0(n) X 0(1), U(n, 1)/U(n) x U(l), Sp(n, 1)/Sp(n) x Sp(l) 

(1-2) 

the analogy of an expansion in the Jacobi polynomials is the index hypergeo- 
metric transform (see |2H) . |1H) ^ 

1 1"°° 

5(5) = T^7^—^ / /(^) 2Fi{h + is,h-is-h + c--x)x''+'-\l + x)''-'dx. (1.3) 
r(fo + c) Jo 
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The problem of decomposition of on these spaces is reduced to the inversion 
formula 



TTT{b + c) 



1 




g{s) 2Fi{b + is,b — is;b + C-, —x) 



T{b + is)r{c + is) 2 
r{2is) 



ds 



for this integral transform. 

Nevertheless, these two classical constructions (i.e., the expansion in the 
Jacobi polynomials and the index hypergeometric transform are not sufficient 
in the analysis on pseudo-Riemannian symmetric spaces of rank 1 (this class of 
spaces includes, in particular, other real forms of the spaces (1.1)-(1.2)). This 
force to think that there exists some another analog (or analogs) of the index 
hypergeometric transform (1.3). We construct one such transform^ ^. 

1.2. The problem on vector- valued bases. The Askey- Wilson hierar- 
chy of hypergeometric orthogonal polynomials is well known, see for instance, 
or PP, Chapter 6. Almost all these polynomials (probably all) appear in 
the representation theory of the group SL2(R) (see, for instance, PS I -P ^ I. 

Consider the tensor product of a unitary highest weight representation of 
SL2(M) and a lowest weight representation. In each factor, there is a canonical 
orthogonal basis consisting of SO(2)-eigenfunctions. Hence there exists a canon- 
ical basis in the tensor product. This basis consists of continuous dual Hahn 
polynomials ( 34 ). Recall (see T, 6.10, JI]) that they are the polynomials 
Pn{s'^) orthogonal with respect to the measure 



But the same problem has sense for each pair of unitary representations of 
SL2(K) or its universal covering. These tensor products (generally) have multi- 
plicity 2 (see |2S], El)- Hence, we must obtain some orthogonal bases consisting 
of C^-valued functions. 

iMolchanov IT7 | . ITsl (19) uses another method. Below we introduce the hypergeometric 
differential operator (2.1). ft arises if we restrict the Laplace operator in tensor product of two 
unitary representations of the universal covering group SL2(Ii)'" of SL2(IR) to eigenspaces of 
the rotation group SO(2). Molchanov |17l (he consider only the group PSL2(K)) restricts the 
Laplace operator to functions that are invariant with respect to the group of diagonal matrices. 
As a result, he obtains the Legendre differential operator on some contour containing singular 
points of the Legendre equation. 

^It is interesting to understand, is it possible to give another proof of the Molchanov's 
Plancherel formula II8I for rank 1 symmetric spaces G/H using the classical index hyperge- 
ometric transform (in the extended variant 0, XIII or and the double index hypergeo- 
metric transform constructed below in 1.4. Precisely, let K he a, maximal compact subgroup 
in G , let y be an irreducible K-module. Consider the action of the Laplace operator on 
HoiUfciV, (G/ H)). Is it correct, that in all the cases we will obtain the hypergeometric 
operator (2.3) on the segment [0,1] (the classical case) or on the contour Hez = 1/2 (our 
case)? 



r{a + is)r{b + is)T{c + is) 2 
T{2is) 

on the line s € K; the explicit formula is 



ds 



Pnis^) = (a + 6)n(a + c)„ 3F2 



—71, a + is, a ^ is 
a + b,a + c 



(1.4) 
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1.3. Notations. We use the standard notations for the Pochhammer sym- 
bol 

{a)n = a{a + 1) . . . (a + n - 1) 
and for hypergeometric functions 



2Fi[a,b;c;z] : = ^ 



3^2 



a, b, c 
d,e ' 



E 



(c)„n! 

(a)n(&)n(c)„^„ 



{d)n{e)nn\ 



1.4. Double index hypergeometric transform. Fix ^ a ^ 1/2, 
P eR. Assume that a + ip ^0. 

Consider the space of C^-valued functions of the half-line s > 0. An element 
of this space can be considered as a pair of scalar- valued functions (<pi(s), (p2(s))- 
Let us introduce the scalar product in this space by the formula 



+ r2i{s)'fi2{s)ipi{s) + r22(s)¥'2(s)V'2(s) 



ds 



J|r(2zs)|2' 



where nj (s) are given by 



r2i{s) r22{s)J 

r(i — a — is)r(i — a + is) r(i — if] — is)T{^ — i/3 + is) 
^r(| +il3- is)r(| +il3 + is) T{1 + a - is)r(i +a + is) 

It is convenient to write this scalar product in the vector form 



(1.5) 



((¥'i,<P2),(Vi,^2)) = ^ 1^°^ (Ms) Ms))R{s) 



ds 



'2{S)J |r(2zs)|2 

Thus we obtain the Hilbert space of C-^-valued functions. We denote it by i?a,/3- 
Let X e M, s > 0. Consider two functions Qi{a,(3;x,s), Q2{cx, f3;x,s) given 
by the formulae^ 

Ql{a,P;X,s) = ^,^-1^(1 +,.x)-("+^«/2(l _^^)(a+i/3)/2-i.-l/2^ 



1/2 - a + is, 1/2 - i/3 + is _ ix + 1/2 
1-a-i/? 'ia;-l/2 



; (1-6) 



^Qi, Qi are almost L^-eigenfunctions of the diflerential operator D defined below (2.1). 
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Q2{a,(3;x,s) = Qi{-a,-(};x,s). 
For a function / e L'^{^), we define the pair of functions <^i(s), <P2{s) by 

/oo 
f{x)Qj{a,(3;x,s)dx, j = 1,2. (1.7) 
-OO 

Theorem 1.1. Let ^ a 1/2, a + 1/3 ^ 0. 

a) The operator f i-^ {(pi,(p2) is a unitary bijective operator from L^(M) to 

b) The inversion formula is 

{QM,P;x,s) Q2(a,/3;:r,.))i?(s)(^jg)^^. (1.8) 

Remark. Let A{s) be a 2 x 2-niatrix valued function on R. Any calibration 

{Ql{s), Ql{s)) := (Qi(s), Q2{s))A{s)-\ R°{s) := A{syR{s)A{s) 

gives another form of our integral transformation. It is interesting to find ma- 
trices A{s), for which the matrix R°{s) and the vector {Qi{s),Q2{s)) remain 
relatively simple. 

1.5. Generalization for a > 1/2. Now, let a > 1/2, /3 G R, and a + 

i(3 ^ 0, 1, 2, Denote by n the integral part of a — 1/2. Consider the finite 

dimensional linear space Wa,0, consisting of vectors (cq, ci, . . . , c„). The scalar 
product in Wa,i3 is given by 

1 " 2a - 2/v - 1 _ , 
(^'^) = ^Z.r(2a-fc)fc! 

Next, define the functions* 

R{a, P; x; k) := 



r(a + i/3)^2+*^^ U 



-fc.fc- 2a+ 1 1 
l-a-i/3 ;2+*^ 



(1.9) 

Now, consider the linear operator 

Ja,fS : L^(R) ^ if«,/3 e W^a,/3, (I-IO) 

given by 

where (fi, ip2 is defined by (1.7) as above, and the coordinates of the vector 
6 € Wa^ff have the form 



Sk = f{x)R{a,P;x;k)dx. 



^These functions are L^-eigenfunctions of the differential operator D defined below 
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Theorem 1.2. The operator Ja^p is a unitary bijective operator. 

1.6. Romanovski polynomials. The Romanovski polynomials j^T] are 
the polynomials on M orthogonal with respect to the weight 

(i + (i - dx. 

This weight decreases as and hence it is possible to orthogonalize only 

finite number of power functions 1, x, x'^, .... Romanovski polynomials are 
defined by the formula 

2-Fi [— fc, k — 2a + 1; 1 — a — iP; ^ + ix], 

i.e., they coincide with (1.9) up to an elementary factor. 
Recall that the Jacobi polynomials are given by 

P;'''^(a;) = const - 2Fi[-n,n + + S,d + 1: {I + x)/2] 

We observe that the Romanovski polynomials are analytic continuations of the 
Jacobi polynomials with respect to the superscripts. 

Many orthogonal systems of this kind are known, see |2], CJ-^nij |21|. 
see also [3], for some applications. Since these systems are eigenfunc- 
tions of some second order differential or difference operators, we obtain a good 
collection of spectral problems (and hence this must give a collection of new 
integral transforms).'^ Recently, one of the most complicated problems of this 
kind (related to one of families of hypergeometric 4F3-polynomials from \21\) 
was solved by Groenevelt [5]. 

1.7. Vector- valued bases. Fix parameters < a < 1/2, P,p,q S K. 
We consider the Hilbert space Y{a,/3;q), consisting of C^-valued functions on 
half-line s ^ with the scalar product 



1 



Ms) 
Ms) 



r(l + iq + is)r(l -iq + is) 



T{2is) 



ds, 

(1.11) 



where the matrix R is the same as above (1.6). 

Next, define functions 'E.n\a, (3]p,q] s), 'E.^^a, (3;p, q; s) in the variable s, 

^It seems, that now only difference operators remain interesting. There are 3 types of 
difference operators related to 

a) Shift operator on the lattice Z: T/(n) = f{n + 1). 

b) Shift operator on the line R: Tf{x) = f(x + 1). 

c) Shift operator on the line R in the imaginary direction Tf(x) = f{x + i). 

It seems, that for the case b), we have infinite multiplicities. In any case, variants a), c) are 
interesting for our purposes. 
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given by the formula 



o cos(p - a - + i/?)7r/2 r(l - a + 

^' - r(a - iP)T{l +iq + if3)T{l + tq - a) 



3F2 



{I - a + ifi - p + iq) /2 - n, 1/2 + iq + is, 1/2 + iq - is^ 
1 + iq + il3,l + iq — a ' 



(1.12) 



^n\a,f3;p,q;s) = ^ll\-a,-P;p,q;x). 



Remark. The hypergeometric series 3^2(01, 02, 02; 61, 62; 1) is absolutely 
convergent for ^ cii < bj and admits an analytic continuation as a meromor- 
phic (sigle- valued) function to arbitrary values of the parameters ai, 02, 02, 61, 
62- We understand (1.12) as a value of this meromorphic function. Below (3.4), 
we represent the function S^^^ as a hypergeometric series that converges for all 
interesting for us values of the parameters. 

Theorem 1.3. The system {'E.n \ sl^^), where n ranges in Z, is an orthonor- 
mal basis of the Hilbert space Y(a, /?; q). 

Remark. Emphasis, that the expression (1.12) has the structure 



const • 3F2 

with 



-ri + h,a + is, a — is ^ 
a + b.a + c ' . 



(1.13) 



Rea = l/2, Re6=:l/2, c e R. 

Also, in comparison with formula (1.4) for the Hahn polynomials, the parameter 
n is shifted by some complex value h. 

Remark. Similar perturbations exist for other classical hypergeometric sys- 
tems (Laguerre, Jacobi, Meixner-PoUachek, Meixner), they are discussed in 

1.8. Further structure of the paper. Theorems 1.1-1.2 are proved in 
§2, Theorem 1.3 is obtained in §3. 

In Addendum we discuss perturbations of Laguerre, Meixner, Meixner- 
Pollachek, and Jacobi systems. 

§2. Spectral decomposition of the hypergeometric differ- 
ential operator on the contour Re 2; = 1/2. 



A proof of Theorems 1.1-1.2 proposed below is direct but very tedious. 
We directly apply the Weyl-Titchmarsh-Kodaira theorem for an appropriate 
differential operator. This theorem with lot of examples is analyzed in Chapter 
XIII of Dunford-Schwartz's book [Sj, see also the Titchmarsh's book |31j . 

^An attempt to find some ways for vector- valued special functions starting from spherical 
functions is contained in (SOi . see also further references in these works. This approach 
differs from our standpoint based on spectral problems with multiple spectra. 
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For the classical index hypergeometric transform (1.3) several proofs of the 
inversion formula are known, see 231> ESli HHi 7.8.8. It is interesting to find a 
shorter proof of Theorem 1.1. 

2.1. Hypergeometric operator. Let a ^ 0, /3 G M. We consider the 
differential operator 

.1 2N_^ r, _^ (g + if3)^ (g - i/3)^ 1 

W+ + 4(1/2 + zx) +4(1/2 -ia;) +4- ^^^^ 

This operator is formally self-adjoint in L^(M). Its resolvent {D — X)~^ is explic- 
itly evaluated below. It is well defined for Im A ^ 0. Hence the deficiency indices 
of D are 0; this implies the self-adjointness of the operator D. Our purpose is 
to construct the spectral decomposition of the operator D. 
Let 

r(x) := (i + i2;)"("+*'3)/2(i - ia;)-*"-'^/^. 
Evaluating directly the differential operator 



we obtain 



Bf:=r-'Dirf), 



'A ' dx^ ^ ' dx ^ 2 

Passing to the complex variable 



z = 1/2- 



we obtain the operator 



A := -z{l - z)^ -{l-a-^(3-z{2- 2g))^ - (g - i)'. (2.2) 
Hence the equation Af — f transforms to the hypergeometric equation 

z(l - z)^ + (c - (a + 6 + l)z)^ -ah\j^O (2.3) 
dz^ az J 



with 



i — g + /i, 5=i — a — /i, c := 1 ~ a — ijS 



Now, we can use the Kummer series for solutions of the equation Df = ^^f. 
Below we use the standard notation ui,. . . , uq of 7^ for the Kummer solutions 
of the standard hypergeometric equation (2.3) and use explicit formulae [J], 
(2.9.1)-(2.9.24) for their expansions in series. 

2.2. Bases in the space of solutions of the hypergeometric equation. 

We will use four bases in the space of solutions of the equation {D — ^j?)f = 0. 
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Basis Si, 82- Writing the Kummcr solutions ui, of the usual hypcrgc- 
ometric equation (2.3), we obtain the following pair of solutions Si, S2 of the 
equation (2.1): 



1/2 - a + /X, 1/2 - i/3 + n ix + 1/2 
1-a-i/? ' ia;-l/2 



X 2F1 



; (2.4) 



S2{a,f3;ii;x) = Si{-a,-(i; ii;x). 

The hypergeometric series in (2.4) is absolutely convergent for \mx > 0, or 
equivalently, for Rcz < 1/2. On the line x S R the series is conditionally 
convergent. Nevertheless it admits analytical continuation through the line 
Ima; = 0, and it is more convenient to think that we consider its analytic 
continuation. 

Further, assume that for x = iy with —1/2 < y < 1/2 (or, equivalently, for 
< ^ < 1) 

(i + ix)^ := (1 _ -^y g.ln(l/2-ix)_ 

Now we can assume that our solutions 6*1, ^2 are defined in the domain {Re 2: < 
1/2} \ [-00,0). 

The solutions ^i, S2 have asymptotics 

5i(x) ~ (i + ^x)("+^'')/^ S2{x) ^ (i + zx)-^''+'^^^'; for x - i/2. 

(2.5) 

We also can define by the formula 



(1 + ix)-^''+'^^/^ (i - ix)-^"-'"^^^ 2F1 V - - « - V - - " - p + 
'^■i ' \i I L V — a. — ip 

(2.6) 

Here the hypergeometric series converges for |l/2 — ix\ < 1/2, i.e., we do not 
obtain an explicit formula on the whole line x € M. 
We mention that 

Si{a,0;-iJ.;x) = S\{a, (3; ii;x). 

Basis Ti, T2. Writing the pair of the Kummer solutions U2, uq, we obtain 
the following pair of solutions Ti, T2 of the equation (D — /x^)/ = 

Ti,2(a, /3; n; x) := Si,2{a, -/?; fj,; -x). 

These solutions are defined in the domain {Re 2; ^ 1/2} \ [1, 00). 

Bases V-, V+ and W-, W+. The Kummer solutions U3, Ui are defined 
outside the circular lune |z|<l, |z — l|<l;in particular, this hmc contains the 
segment —V3/2 < x < •\/3/2. Hence the analytic continuations of U3, from 
the points z = 1/2 + ioo and z = 1/2 — ioo to the whole line Re 2 = 1/2 are 
different. This gives the following two pairs V±, W± of eigenfunctions of (2.1). 



1/2- a + fj.,1/2 - a - IJ,_ . 
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We define the solution V_ of the equation {D~ fi^)f = as the solution that 
for X > \/3/2 is given by the formula 

V. {a, /?; fi; x) = e(-i/2+a+./3-M)W2 ( i ( i -i/2+("+^/3)/2-m ^ 



X 2F1 

Further, assume 



l/2-a + ^, l/2-i/3 + /^. 



1 + 2/z ' 1/2- ix 



(2.7) 



V+{a,^3]^l]x) = VL(a,/3;-^;a;). 
The asymptotics of V± for x +00 has the form 

V± - X +00. (2.8) 

The solutions W± are determined by the condition 

iy± ~ x^/^^'", a;->-c». (2.9) 

To obtain a formula for W- for x < we must change the sign in 

the argument of the exponential function in (2.7). Next, W+{a, P; fi] x) = 
W-{a,P;-^i;x). 

2.3. Transition matrices. Define the constants 

r( R \ r(a + »/3)r(l + 2M) 

r(l/2 + a + ^)r(l/2 + ip + /i) 
X(a, /3; A<) := e-(i/2+«+»/3-M)W2, (2.11) 

In this notations (we use |7|, (2.9.37), (2.9.39)), 

(a, 13] fi; x) =C(a, /3; fi)xi(^, P; f^)Si {a, (3; fi; x) + 

+ C(-a, -/?; M)x(-a, M)^2(a, /3; fi; x); (2.12) 
V+ (a, /?; ^; x) =C(q!, /3; -n)x{a, P; -f^)Si (a, /?; pi; a;)+ 

+ (^(-a, -/^; -Ai)x(-a, -M)'5'2(a, /?; p^; a;)- (2.13) 

Formulae expressing W+ in 6*1, ^2 can be obtain from (2.12)-(2.13) by 
the transform x ^ X~^- 

We also need in an expression of W± in terms of Ti , T2. Similarly, applying 
13, (2.9.38), (2.9.40), we obtain 

W- (a, /3; fi;x) =- C{a, -(3; fi)x~^{-a, /?; -n)Ti{a, /3; fj.; x)- 

- C{-a, p; ii)x-\c, -P- -ti)T2{a, P; /x; x) (2.14) 

To obtain a formula for W+ we must change sign of /i in C{a,P]ii) and in 

x{a,P; m)- 

P Q 
,P' Q' 

the Wronskian of two arbitrary solutions P, Q of the equation {D — ii^)f = 
This expression must have a form 



2.4. Evaluation of Wronskians. Denote by wr(P, Q) :— det 
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where a{P, Q) is a constant, see ^U]) 1-17.1. 

The value (j{Si^S2) can be easily evaluated using asymptotics (2.5), we 
obtain 

cr(S'i, S'2) = i{a + 

Below, we need in aiV-, W-). The determinant A of the transition matrix from 
the basis [Si, S2) to the basis [V-, W^) can be easily evaluated, this gives 

CT y_,Ty_ = A • cr(S'i,S'2 = — — — . , , —s — ■ (2.15 

1 (a + ip)L {—a — ip) 

2.5. Kernel of resolvent. Now we are ready to write the kernel K{x, y; A) 
of the resolvent 

i?(A) := (D-X)-' 

of the operator D, 

R(X)f{x)^ / K(x,y;X)f{y)dy. 



Assume A e C \ (—00, 0). Then the solution V- is an element of L^(0, +00) and 
W- e L2(o,_oo), sec (2.8)-(2.9). Hence (see jSi,XIII.3.6), 



K{x,y;X) 



a{V.,W.) ' 
V.{y)W.{x) 



a{V-,W-) 



for X < y, 



the value (t(VL,T4^_) was evaluated above (2.15). 

We intend to write an expansion of the differential operator R in eigenfunc- 
tions. According prescription rising to Weyl's work ^33' (see P3 and detailed 
presentation in Dunford, Schwartz XIII. 5. 18), we must evaluate the jump of 
the resolvent on the real line 

— lim / (R(X + ie)-R(X-ie))dX, (2.16) 

27716^0 J _^ 

and represent it in the form 

/oo , poo 
(/ a,{x,X)a,{y,X))fiy)dfi,,iX))dy (2.17) 

for some solutions cti, cr2 of the equation Df = A/. Here lJ-ij{X) are (complex- 
valued) measures on K. Then /i^ is the spectral measure. Precisely, the operator 

f{x)'yi{x)dx, I f{x)a2{x)dx^ 
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is a unitary operator from L^(IR) to the space of C^-valued functions {ipi (A), <^2(A)) 
with the scalar product 

/oo 
v9.(A)^d/x,,(A) 

(in particular, the right-hand side of this equality is a positive definite scalar 
product). 

2.6. Formula for resolvent. To evaluate the jump of the resolvent for 
A ^ and for A < 0, we are need of two explicit expressions for the kernel 

K[x, y; A) of the resolvent. 

Expressing V- via Si, S2 and W- via Ti, T2, we obtain the following ex- 
pression for the resolvent 



K{x,y;X) = ^T{a+il3)T{-a-il3) 



C(a,-/3,VA) 
C(-a, V A) 



C(-a, -/?; V A) C(a,/J;VA) 



C{-a,P; V\) 
C(a,/3; VX) 



e(-V2-«-^^'^'52(a;)T2(y)l, (2.18) 



where C(. . . ) is defined by (2.10). 

Expressing VI, W- by 5i, 6*2, we obtain 

/.(.,,; A) ^ r(. + .-£)r(-.^./?) 

2TTC{a,(3;y/X)C(-a,-/3;y/X) 
X C(a,/3; VA)) x(a, /?; VA) 5i(x) + C(-a, -/3; VA)) x(-a, -/3; VA)) S2{x) 
C{a, (3; %/A)) x-'(a, P; ^) Siix)+C{-a, -/?; v/A)) x"'(-a, \/A)) 52(x 



(2.19) 



where x is given by (2.11). 

These two expressions are meromorphic in the plane A e C, with a cut on 
the negative semiaxis A < 0. 

2.7. Jump of resolvent for A > 0. Evaluation of 

1 f°° 

^[0,00) ._ / _^ _ _ /2.20) 

2ttI e^O Jq 

is reduced to an evaluation of the residues. Expression (2.19) has simple poles 

at the points A = (a — k — 1/2)^ for integer k, satisfying the condition ^ fc ^ 
a — 1/2. In addition, only the coefficient at Si{x)Si{y) has nonzero residues, 
three other coefficients are holomorphic at these points. 
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Thus the integral operator (2.20) equals 

L[0.-)f(:r) = -r(a + */3)rfa-z/3) V (2^ - 2fc - 1) (1 - a - 

/W ^^i^a + ip)i{a ip) 2^ r(2a-fc)(l-a + z/3)fe/fc! 

Si{a,/3;a- k - l/2;x)Si{a, /3;a - k - l/2;y)f{y) dy 

Writing 5*1 at the form (2.6), we obtain its representation in the terms of the Ro- 
manovski polynomials. Applying identity [7], (10.8.16) for Jacobi polynomials, 
we convert our expression to the form 



1 2a — 2fc — 1 

-T{a + z/3)r(« - z/3) r(2a-fcU' ^^ (a, /3; a - fc - 1/2; x) x 



2a -2k -I 

0^fc<Q-l/2 



Si{a, P;a-k - 1/2; y)f{y) dy 



and this gives the required expression. 

2.8. Jump of resolvent for A < 0. We must evaluate 

K{x,y;VX)-Kix,y;-VX), 

assuming that A is negative real and Im -^A > 0. 

For definiteness, we present the calculation of the coefficient at Si{x)Ti{y). 
In three remaining cases, the evaluation is almost identical^. 

Thus, (see (2.18)), we intend to find 

^ r{a + ip)r{-a-ip)x 



-27r 



^ J- C{a,-(3;VX) ^-(i/2-a+x/A)„ _ C(a,-/3;-VA) ^-(i/2-a-V\)^t\ 
I C(-a, -/3; VA) ^(-a, -/?; -\/A) J' 

After direct cancellations, we obtain 
-^r{a + iP)T{a - i/3)x 

— ZTT 

^ f r(l/2-^WA)^_(i/2_„+^)„ ^ nl/2-a^VX)_^-(^l/2-a-^).^^ 
'-r(l/2 + a + \/A) r(l/2 + a-VA) J' 

We convert this expression to 

— i-r(a + iP)r(a - i/3)r(l/2 -a + VA)r(l/2 - a - \/A)x 
— 27r 

, „-(l/2-Q+\/A)7ri p-{l/2-a-V\)ni 



■r(l/2 + a + ^/X)T{l/2 - a - \/A) r(l/2 + a - VA)r(l/2 - a + %/A) ■ 



^This is explained by natural symmetries {a, /3) ~ (— o, — /3) ^ — ia) of the equation 
(2.1). 
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Now, we will transform only the expression in the curly brackets 



. . } = cos(7r(a + ^/A))e-(l/2-«+^)-' - cos(7r(a - \/A))e-(i/2-«-^)-. 

Next, we apply the Euler formulae for cos, collect similar terms in the linear 
combination of exponential functions, and again apply the Euler formulae. We 
obtain 

(2\/A 

and this finishes the evaluation of the coefficient at S\{x)T\(y). 

After this, for the jump of the resolvent on the semiaxis (—00,0], 

we obtain the expression of the form 

L^-oo,^)^{x)= E /" A^iA^) I Si{x)Tj{y)f{y)dy\ 

where Oij are some explicit products of F-functions. 

Further, observe that for A < and y e R we have Tj (y) = Sj (y) . We obtain 
an expression of the form (2.17), and this finishes the evaluation of the spectral 
decomposition of the operator (2.1). 

The parameter s from 1.4 is VX. 

§ 3. Construction of bases 

3.1. One basis in L^(R). Fix real parameters p and q. 
Lemma 3.1. The system of functions 

where n ranges in Z, is an orthonormal basis of L'^{R,dx/ 2Tr). 
Proof. Pass to a new variable ip e [0, 2tt] by the formula 



1/2 -ix' ^ l/4 + a;2' 
Our system of functions transforms to 

g-mv> . e-'P*/2(2cos^/2)'9. 

We obtain the standard orthogonal system e~^'^^ up to multiplication by a 
function whose absolute value; is 1. □ 

Emphasis also that orthogonality of our system can be obtained directly 
(since the scalar products can be easily evaluated using residue calculus). 

3.2. Construction of bases. The bases (1-13) can be easily obtained, if 
we apply the double index hypergeometric transform to the orthogonal system 
(3.1). 
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Let us explain, how to perform the calculation. We must find 

rl"^ {x)Qi{a,(3;x;s) dx 



(3.2) 



where the function Qi is defined by the formula (1.6). For this purpose, we 
expand 2^1 from formula (1.6) in the hypergeometric series in powers of 

ix + 1/2 
ix - 1/2 

Then we integrate it termwise using the Cauchy beta-integral (see |26| . v.l, 
(2.2.6.31) or [T], Chapter 1, ex. 13) 



dx 



2nT{a + T - 1) 



(3.3) 



, (1/2 + ixy{i/2 - ixY r(cr)r(T) 

As a result, we obtain the following expression for (3.3) 

27rr(l/2 + ig- is) 

r(a - i[3)T{{p + iq - a + z/?)/2 + n + 1 - is)T{{-'p + iq + a ~ i(3 + 1) /2 ~ n)' 



X 3F2 



1/2- a- is, 1/2 + i/3 - is, 1/2 + [p - iq - a + ij]) /2 + n 
1 — a + i(3, {p + iq — a + iP)/2 + n + 1 — is 



(3.4) 



This expression is a variant of the final answer, its more symmetric form (1.13) 
can be obtained by Thomae transformation see Q], Corollary 3.3.6, [211, v. 3, 
(7.4.4.2). 



V{d)T{e)T{r) 
T{a)T{b + r)T{c + r) 



3F2 



I— a,e — a,r 
b + r, c + r ' 



where r = d + e~a — b — c. 
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Addendum. 
Perturbations of some 
classical hypergeometric orthogonal systems 

Here we discuss orthogonal systems that can be obtained from some classical 
orthogonal system r„ by transformation n t-^ n + 9, there 9 £ C is fixed. 

The most of facts formulated in this addendum are known or semi-known. 

A.l. Spectral problems for one-parametric subgroups in SL(2,R). 
Consider a unitary irreducible representation of the universal covering of the 
group SL(2,R). There are three (up to a conjugation) one-parameter subgroups 
in SL(2,R), namely 

- an elliptic subgroup K = SO (2), consisting of orthogonal matrices; 

- a hyperbolic subgroup D consisting of diagonal matrices ^il ' where 
a > 

- a parabolic subgroup N consisting of matrices 

Consider the spectral decomposition of our representation with respect to 
one of these subgroups (i.e., consider a realization of our representation in some 
space L^, where our subgroup acts by multiplication by functions). Operators 
of representation of SL(2,IR) in these models were obtained in the books of 
Vilenkin Vil , Chapter 7, and of Vilenkin and Klimyk (VKlj, Sections 6.8, 7.2, 
7.68 9 

Also consider some subgroup in SL(2,M) conjugated to S0(2), consider its 
eigenbasis in the space of representation, and consider the image of this eigen- 
basis in our space L^. Obviously, we will obtain an orthogonal basis in the 
space L^. Considering the spectral decomposition of a highest weight represen- 
tation with respect to K, D or N, we obtain respectively Meixner polynomi- 
als, Meixner-PoUachek polynomials and Laguerre polynomials (see jVKl| . 6.8.3, 
7.7.11, 7.7.8). 

Starting from a representation of a principal or complementary series, we 
obtain some nonpolynomial bases. These cases are discussed below in A.2-A.4. 

Remark. In A. 2 - A. 4, we omit representations from our discussion and for- 
mally use only elementary analytic tools. But the representation theory allows 
to propose simple and effective actions. In fact, initial orthogonal bases, that 
we use below, are eigenbases for elliptic subgroups (for representations realized 
in functions on line or circle, see, for instance, |VKlj ). the Fourier transform 
is the spectral expansion with respect to the parabolic subgroup N, the Mellin 
transform is the spectral expansion with respect to hyperbolic subgroup D, and 
expansion in Laurent series corresponds to expansion in eigenvectors of K . 

®To be precise, they usually consider the group SL(2,R) itself, extension of their formulae 
to the universal covering group is not a difficult problem. 

^Likely, the earliest references that can be attributed to this subject are 
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A. 2. Perturbation of Laguerre polynomials. Recall that the Kum- 
mer's confluent hypergeometric function iFi is given by 

cx; , N 
n— 

The confluent hypergeometric function of Tricomi (for details, see IHTl'll .S.S.? 
or |Slalj ) is given 

^{a, c; x) = T^^^^—^iFiia; c; x] + Ei^Z^x^~^iFi[a - c + 1; 2 - c; x] 
1 (a — c + 1) 1 (a) 

Fix a E C Consider the orthogonal basis 

ipn{a;x) = [l/2 + ix) ' {l/2-ix) ' 

m L'^{R), where n ranges in Z. 

Considering the Fourier transforms of these functions, we obtain (see |IT1) . 
III. 3. 2. 12; see similar integrals in |Slal,) the following proposition. 

Proposition A.l. The functions £j^-^{y) onR given by 



— - — — *(l/2 + n + a,l + a-a;y), for y > Q 

1 (—71 + 1/2 — a) 

— — — ^*(l/2-n-a,l+S-a;-zj), fory<Q 
Tin + 1/2 + a) 

form an orthonormal basis in on M with respect to the weight exp(— |y|). 
Now, consider a, /3 e M, let 

|a-/3|<l/2, and a, /3 7^ ±1/2. 

Define a weight fj,a.f3{y) dy on the line R by the formula 

ra,p\yj \y\ [cos(7ra), for y < 

Define functions H"'^ on M by 
1 



*(l/2 + n + /3, l-Q; + /?;y), for y > 



r(l/2 -n-a) 

— — — *(l/2-n-a,l-a + /3;y), for y < 
r(l/2 + n + Pj 



Proposition A. 2. The functions ^"'^ > where n range in Z, form an or- 
thogonal basis in the space L'^ (K, fi(y) dy) . Moreover 

\K''^iy)\^Ky)dy - 

J " ^^^1 ^ r(l/2 + n + /3)r(l/2-n-a) 
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To prove orthogonality, we write explicitly the orthogonality relations for 
and apply the Kummer formula 

*(a, c; x) = x^^'''i'{a - c + 1, 2 - c; x) 

After this, we write the analytic continuation of orthogonality relations with 
respect to a, a. 

Proposition A.l corresponds to the principal series of representations of 
SL(2,R) and Proposition A. 2 corresponds to the complementary series. The 
cases a = 1/2 and 13—1/2 correspond to highest weight and lowest weight rep- 
resentations. In these cases, the system L'^'^ degenerates to the usual Laguerre 
(Sonin) system on half-line. 

On these functions £;„, see Groenevelt |Gro| . 

A. 3. Perturbation of Meixner polynomials. We use the following 
nonstandard notation for the Gauss hypergeometric function. 

.Fl [a, 6; c; .] := .F, [a, 6; c; .] = g ^(Jfc)fc, ' 

If c = — / = 0, —1, —2, . . . , when 2-^1 [a, c; z] has poles, but 

^-^ ml (m + /)! 

m— ^ ^ 

= — ^ 2Fi[a + l + l,b + l + \\l + l\z\ 

(first (I + 1) summands in the series for 2F1 are zero). 
Consider the following orthogonal system 

K{z) := z"(cht - zsht)-i/2-«-n(cht - z-^ sht)-i/2+"+" 

in on the circle \z\ ~ 1; here n ranges in Z. Expanding these functions 
in Fourier (Laurent) series, we obtain the following orthogonality relations for 
their Fourier coefficients. 

Proposition A. 3. For fixed a e C, t e M, the functions 

(thty-^ rl/2 + a + J, 1/2 - « - n. ,,2 



M^'"(j) = \ '_ 2F{ 

1 (1/2 — a — n) 



, th^ t 
J - n-f 1 



where n ranges in TL, form an orthogonal basis in I {"L). The inner squares of 
all the elements of the basis are 

Proposition A. 4. Fix t eR, fix a, p eR such that 

\a\ < 1/2, |/3| < 1/2 
Consider the space Ha,p of two-side sequences x{j) with the scalar product 
I N ,-^-r^ r(l/2-Ka + j) 
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Then the system of functions 

y^-'^(,^ - ft^^V'" p* [1/2 + a + j,l/2^a- n. 2 , 
^" (^)-r(l/2-/3-n) j-n-1 ' \ 

forms an orthogonal basis in Ha,f}, and 

n r(l/2 + /3 + j) ^ ' ' 

Proof. Consider the following biorthogonal system on the circle 

u„(z) = z"(chi - zsht)-^^^-"-'\cht - z-^ sh<)-i/2+/3+"; 
Vn (z) = z" (ch t - z sh t) -1/2-^-" (ch t - z- 1 sh i) - 1/2+a+n 

Expanding these function in Fourier (or Laurent) series, we obtain a biorthogo- 
nal system of two-side sequences. Images m„, Vn of u„, ?;„ seem different at the 
first glance. But applying the identity (see [HTFlj . (2.1.23)) 

2F1 (a, 6; c; z) = (1 — z)''^"^^ (c — a, c — 5; c; z) 

we observe that u„, t;„ coincide modulo a simple elementary factor. Thus we 
obtain orthogonality relations for m„. □ 

These facts are contained in Vilenkin-Klimyk |VKlj (they discuss only two 
parametric family of bases) , apparently, these bases firstly appeared in jVKOj . 

A. 4. Perturbation of the Meixner— Pollachek system. Consider the 
Mellin transform in L-^(M), i.e., for g S L^(M), we define two functions 



gi{s) / x^'-'/^g{x)dx 
Jo 

52(5):=/ {~xy^-'^'g{x)dx 



on R. We have the following condition of unitarity 

\g{x)\ dx^—^J |?i(s)| dx + J |g2(s)| dxj 

Fix a = (J + ir E C and (p E (0, tt). Consider the orthogonal basis in L^(R) 
given by 

h„{x) = {1 + a;e*'^)-i/2-'^-i^-"(i + 

We evaluate Fourier transform of these fimctions (see |PBM1| . 2.2.6.24) and 
obtain the following construction. 
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Consider the Hilbert space Va, whose elements are pairs of functions (/i, /2), 
depending in the variable s € M, and the inner product is 

,w ^\ |r(i/2 + ZT + z.)r(i/2 + iT- 

((/l,/2), (51,52)) = TT- I O- M2 ^ ^ 

^ ' 2 sin(p|l (1 + 22Tjp 

J —oo J — 00 

Consider the hypergeometric series 

l/2 + is + iT,l/2~a + iT-n^ 



2 



l + 2lT ' 



We consider values of analytic continuation of this series at the point 

u — I — e~'^^'^, where < tp < it 

By G\{s, ip) we denote the value obtained after passing the way h{8) := 1 — e^^*^, 
where 9 e [0, (^]. By G'^{s](p) we obtain the result of passing h{0) := 1 — e^*^, 
where € [Q.tt — Lp\. 

Proposition A. 5 The system {G}^{s) , G^{s)) is an orthonormal basis of the 
space Va- 

Thus, we obtain a C^-valued orthogonal system (as above in Section 3). 

A. 5. On properties of perturbed systems. Standard properties of 
Laguerre, Meixner-PoUachek, and Meixner polynomials (see |HTF2| or |KSp 
can be easily extracted from the theory of highest weight representations of 
SL(2,M); this is explained in Vilenkin-Klimyk |VK1| . It is clear that their 
methods remain valid for the perturbed systems. 

One of nice facts that can be obtained in this way is surviving of the Meixner 
generating function (see |Mei| \ . For instance, consider the Poisson kernel for 
perturbed Meixner-PoUachek system. 



n— — 00 



^" 2. [Glis)GUt) Glis)GUt))' 



This sum admits an explicit evaluation, since 3C(s, t; 9) are the kernels of integral 
operators corresponding to matrices 

/ cos6' sin6'\ ™^ 
g{9) = [ . „ „ e SL(2,R) 
' y—sm9 cos 9 J ^ ' 

in the D-diagonal model of the representation. The latter kernels (for 2ct G Z) 
were evaluated by Vilenkin [Vil , (7.4.1. 13)-(7.4.1. 16) (see also |VKT) . (7.2.1.12)- 
(7.2.1.15)). 

The same is valid for the perturbed Meixner system and perturbed Laguerre 
system (for the Laguerre system, this gives an extension of Myller-Lebedeff 
bilinear generating function, see jHTF2j . |M-Lp . 
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A. 5. Perturbed Jacobi systems and Molchanov's singular bound- 
ary problems. Let — l<a<l,/3>— 1. Fix 0, such that < 6' < 1. Wc also 
assume < 6* + a < 1 . 

Let n — 9 be integer. We consider a function <f>„ on the half-hne x > given 

by 



-n, n + a + P + 1 



(3+1 

X 2F1 



n + a + P + 1, a + n + 1 
2n + a + [3 + 2 

n + a + f3 + l,2n + a + l^ 1" 
2n + a + f3 + 2 ' x. 



for < x < 1 



for X > 1 



Remark. These two functions are Kummer solutions jHTFl| . (2.9.11), 
(2.9.14) of the same differential hypergeometric equation. 

Theorem A. 6. The system $„, where n - 9 e'L and 2n + a + /3+l>0, 
is orthogonal with respect to the weight 



w{x) 



xl^{l-xT, 

s\\\{n9) 
sin7r(a + 1 



-x^{x-iy 



for < a; < 1; 
for X > 1 



and 



\^n{x)\^dx 



r2(/3+l)r(l + n)r(l + n + a) 



{a + l3 + 2n+ l)T{[3 + 1 + n)T{n + a + {3 + I) 

Remark. We emphasis that the orthogonal system $„ is not a basis. 
Proof. Denote by H{x) the Heaviside one-step function, H{x) — 1 for 
a; > 0, and H{x) — for a; < 0. Consider two Mellin-Barnes integrals 



/3 + 1, 1 + m + a 
f3 + m + l 

7? + 1 + m, —TO — a 
(3+1 



s, P + l + m — s 
m + a + l + s,(3+l~s 



'ds = 



= 2F1 



;x 



H{l-x) 



-P-l-m-r 



P + 1, 1 + m + a 
-m,2m + a + (3 + 2 



■/3 + 1 + TO, TO + 1^ 1 

. 2m + a + (3 + 2 'x 



H{x-1) 



- 2^^ 



'2n + a + (3 + 2, -a-n 
n + a + (3+1 



+'ioo 



2F1 



n + a + (3 + 1, a + Ji + 1 



2n + a + (3 + 2 

'2n + a + (3 + 2,-a-n 
n + 1 



2F1 



a + n + s, (3 + 1 — s 
s, n + P + 2 - s 

H{l-x) + 



'n + a + P + l,n+l 1" 
P + 1 'x. 
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These formulae can be easily checked using the standard technic of the Mellin- 
Barnes integrals, see |Sla2| (see also tables |PBM3j . 8.4.49). 
We intend to evaluate 

Jo ^x' X 

(this integral is the inner product of <i>„i and <&„ up to a F-factor). 

Since the Mellin transform transfer a convolution to a product, we must 
evaluate 



27ri 



s, /? + 1 + m — s 
m + a + l + s,f3+l — s 



a + n + s, P + 1 — s 
s, n + p + 2 - s 



ds ~ 



1 
2^ 



+ioo 



/3+1 + m — s, a + n + s 
TO + a + l + s,n + /5 + 2 — s 



ds 



= r 



a + f3 + n + m + l, 1 
a + f3 + n + m + 2, l + m — n, 1 + n — m 



(see jPEM3p . The final expression is zero if m 7^ n and m — n G Z. 

Remark. Let us explain why our calculation is not artificial. Let us evaluate 
the inner product of two Jacobi polynomials 



2F1 



-n,n + a + P + 1 
P+l ' 



-m, m + a + /3 + 1 
+ 1 ' 



x'^{l-x)"dx 



in the following way. We represent the integral as a multiplicative convolution 
of 



Ki{x) = (1-.T)" 2-Fl 



-m, m + a + P + I 

-n,n + a + P+l 1 
P+l 'x 



H{1 - x) 
H{x - 1) 



and evaluate the convolution using the Mellin-Barnes integral representations 
of Jacobi polynomials (see PBM3 , 8.4). The calculation obtained in this way 
admits analytic continuation to noninteger number n of "Jacobi polynomial" 
Pn''^ ■ This gives the required formula for if 1, 7^2- D 
Orthogonal systems of this type appear in the following situation. Consider 
the Laplace operator A on a pseudo-Riemannian symmetric space G/H. Re- 
stricting A to the subspace of ii-invariant functions, we obtain an ordinary 
second-order differential operator. This operator is the usual hypergeometric 
operator (2.3) on some contour passing through one or more singular points of 
the hypergeometric equation (different series of symmetric spaces give differ- 
ent contours). Molchanov obtained explicit spectral decomposition for several 
boundary problems of this type (see |Mollj . |Mol2j ). These singular differential 
operators have countable discrete spectra and hence we obtain (noncomplete) 
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orthogonal systems consisting of piece- wise Gauss hypergeometric functions. 
Our construction is not covered by such examples and also does not cover them. 

A. 6. Perturbation of Wilson system. Koornwinder jKooj observed that 
the Wilson 4F3 orthogonal system can be obtained by application of the index 
hypergeometric transform to Jacobi polynomials. It is possible to apply this 
method to a perturbed Jacobi system. These gives (noncomplete) orthogonal 
systems, whose elements are pairs of 4_F'3-functions. 

A. 7. An example of 2-F2-basis. Since we discuss possibilities of obtaining 
of hypergeometric bases using standard integral transforms, let us construct a 
basis that is not a deformation of a classical system. 

Fix p > — 1. Consider the function r„ on M given by 

f (1 - a;)''/2pp^0(a;) for -1< a; < 1 
I 0, otherwise 

where -P^''' is a Jacobi polynomial. Obviously, the functions r„(a; + 2to), where 
n = 0, 1, 2, . . . and to e Z, form an orthogonal basis in L^(M). 

Evaluating their Fourier transforms (see jPBMl| . 2.22.5.1), we obtain the 
following statement 

Proposition A. 7 The functions 



—2miC, TP 



p/2 + 1, l-p/2 
.p/2 + n + 2, -p/2 + 1 - n' 



where n = 0, 1, 2, . . . and m G Z, form an orthonormal basis in L^(]R). 

Our expression 2^2 is relatively simple. Up to a constant (depending on n) 
it equals 

^ (p/2 + l + fc)„ 
^(-p/2+l-n + fc)„fc!^ ^' 

Remark. A toy construction in the same type can be obtained by applying 
the Fourier transform to the standard trigonometric system e™^ on [— tt, tt]. 
Thus the functions sin(y7r)e^"™^/(n + y), where to e Z, n = 0, 1, 2, . . . form 
an orthonormal basis in L^fM). 
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